Learning to Identify Irrelevant State Variables

When they are available, safe state abstractions improve the efficiency of reinforcement learning algorithms by allowing an agent to ignore irrelevant distinctions between states while still learning an optimal policy. Prior work investigated how to incorporate state abstractions into existing algorithms, but most approaches required the user to provide the abstraction. How to discover this kind of domain knowledge automatically remains a challenging open problem. In this paper, we introduce a general approach for testing the validity of a potential state abstraction. We reduce the problem to one of determining whether an action is optimal in every state in a given set. To decide optimality we give two statistical methods, which trade off between computational and sample complexity. One of these methods applies statistical hypothesis testing directly to learned state-action values, and the other applies Monte Carlo sampling to a learned Bayesian model. Finally, we demonstrate the ability of these methods to discriminate between safe and unsafe state abstractions in the familiar Taxi domain.